Evaluating Memory Efficiency and Robustness of Word Embeddings

Skip-Gram word embeddings, estimated from large text corpora, have been shown to improve many NLP tasks through their highquality features. However, little is known about their robustness against parameter perturbations and about their e ciency in preserving word similarities under memory constraints. In this paper, we investigate three post-processing methods for word embeddings to study their robustness and memory e ciency. We employ a dimensionality-based, a parameterbased and a resolution-based method to obtain parameter-reduced embeddings and we provide a concept that connects the three approaches. We contrast these methods with the relative accuracy loss on six intrinsic evaluation tasks and compare them with regard to the memory e ciency of the reduced embeddings. The evaluation shows that low Bit-resolution embeddings o↵er great potential for memory savings by alleviating the risk of accuracy loss. The results indicate that post-processed word embeddings could also enhance applications on resource limited devices with valuable word features.